18-741 Final Report: DBmbench

Conventional database benchmarks, such as the TPC-C and TPC-H, are extremely computationally demanding, pushing databases to their limits as they simulate the operations of real applications. However, this computational intensity demands powerful, costly server hardware, limiting the availability of these benchmarks to only those who can afford such servers. For many researchers interested in exploring new architectural techniques to improve database performance, the computational demands of the TPC-C and TPC-H benchmarks are in direct conflict with the methods of simulation often used in architecture research. Simulated environments provide the ability for researchers to control and modify aspects of computer architecture not typically accessible, such as cache organization, instruction fetch and execution units. Thus, in order to perform an evaluation of a new architecture, scaled-down benchmarks are typically used. While it is assumed that these scaled-down benchmarks exhibit the same micro-architectural characteristics as their original forms, it is unclear whether or not this is the case. Shao et al. [7] have proposed a framework for scaling down the TCP-C and TCP-H benchmarks, while preserving their micro-architectural characteristics. Their primary motivation was to allow these benchmarks to be used in a simulation environment, and their study concluded that their scaled-down benchmarks captured over 95% of the processor and memory performance of Decision Support System (DSS) and Online Transaction Processing (OLTP) workloads. However, their evaluation used real hardware (Pentium III), and thus it still remains unclear whether their scaled-down benchmarks are computationally tractable in a simulation environment. Further, it is also unclear if their results are an artifact of having run on the Pentium III, or if they generalize to other processor platforms. In our work, we extend the results of Shao et al., by running their scaled down benchmarks (μTPC-C and μTPC-H) in a simulation environment. Similarly, we also run the benchmarks on the AMD Opteron architecture. The experiments will allow us to investigate the validity of the micro benchmarks in modeling the full benchmarks for these architecture.